Analysis of Goodreads data on Kaggle

Overview

First of all, we will mention a bit about the owner of this dataset, Goodreads . Goodreads is a social cataloging website that allows individuals to freely search its database of books, annotations, and reviews.

This project aims to have an insight about different numerical parameters of the book and their relationship. Moreoever, alongside with the informaion given, we try to do a bit of web-scrapping and get another attribute (in this case, ‘Year’) in order to take our analysis further. Moreover, this would be my first milestone into Data Analysis, using all the skills and knowledges I have acquired during my summer time working with R.

This project is made possible by user Soumik on Kaggle, along with the wonderful reference from the Python notebook named Goodreads: Analysis and Recommending Books of Shivam Ralli. I would like to send my sincere thanks to both of them.

All of the datasets and scripts used will be uploaded along with this notebook.

Loading libraries and understanding the basics of this dataset

Loading libraries

First, we will load all the necessary libraries for this notebook.

library(ggplot2)
library(dplyr)

Attaching package: ‘dplyr’

The following objects are masked from ‘package:stats’:

    filter, lag

The following objects are masked from ‘package:base’:

    intersect, setdiff, setequal, union

library(RColorBrewer)
library(viridis)  

Loading required package: viridisLite

library(WVPlots)
library(naniar)
library(e1071)
library(plotrix)
library(ggcorrplot)
library(waffle)
library(extrafont) 

Registering fonts with R

library(GGally)

Attaching package: ‘GGally’

The following object is masked from ‘package:dplyr’:

    nasa

library(tidyverse)

── Attaching packages ────────────────────────────────────────────────────────────────────────────────────────────────────────── tidyverse 1.2.1 ──
✔ tibble  2.1.1     ✔ purrr   0.3.2
✔ tidyr   0.8.3     ✔ stringr 1.4.0
✔ readr   1.3.1     ✔ forcats 0.4.0
── Conflicts ───────────────────────────────────────────────────────────────────────────────────────────────────────────── tidyverse_conflicts() ──
✖ dplyr::filter() masks stats::filter()
✖ dplyr::lag()    masks stats::lag()

library(cluster)
library(factoextra)

Welcome! Related Books: `Practical Guide To Cluster Analysis in R` at https://goo.gl/13EFCZ

library(rvest)

Loading required package: xml2

Attaching package: ‘rvest’

The following object is masked from ‘package:purrr’:

    pluck

The following object is masked from ‘package:readr’:

    guess_encoding

Understanding the basic structure of the dataset

We import and investigate the structure of the dataset.

goodreads <- read.csv('/Users/ngohoanganh/Desktop/Goodreads Kaggle project/books.csv', stringsAsFactors = FALSE)
str(goodreads)

'data.frame':   13724 obs. of  10 variables:
 $ bookID            : int  1 2 3 4 5 8 9 10 12 13 ...
 $ title             : chr  "Harry Potter and the Half-Blood Prince (Harry Potter  #6)" "Harry Potter and the Order of the Phoenix (Harry Potter  #5)" "Harry Potter and the Sorcerer's Stone (Harry Potter  #1)" "Harry Potter and the Chamber of Secrets (Harry Potter  #2)" ...
 $ authors           : chr  "J.K. Rowling-Mary GrandPré" "J.K. Rowling-Mary GrandPré" "J.K. Rowling-Mary GrandPré" "J.K. Rowling" ...
 $ average_rating    : chr  "4.56" "4.49" "4.47" "4.41" ...
 $ isbn              : chr  "0439785960" "0439358078" "0439554934" "0439554896" ...
 $ isbn13            : chr  "9780439785969" "9780439358071" "9780439554930" "9780439554893" ...
 $ language_code     : chr  "eng" "eng" "eng" "eng" ...
 $ X..num_pages      : chr  "652" "870" "320" "352" ...
 $ ratings_count     : int  1944099 1996446 5629932 6267 2149872 38872 18 27410 3602 240189 ...
 $ text_reviews_count: int  26249 27613 70390 272 33964 154 1 820 258 3954 ...

head(goodreads)

print(paste('The original goodreads book dataset has',nrow(goodreads),'rows and',ncol(goodreads),'columns.'))

[1] "The original goodreads book dataset has 13724 rows and 10 columns."

Fixing problems of the dataset

First of all, we know that this dataset has 13724 observations with 10 variables. There is one variable that is supposed to be written as “# num_pages”. However, as R cannot read special characters in column names, for the sake of convenience, we rename this variable as “num_pages”.

We obverse some problems with this dataset. At columns 4 and 8 (corresponding to variables “average_rating” and “num_page”), we can see that the observations are of type “character”, while they should be of type “integer”. We fix the problem with the function as.numeric in the following code chunk.

However, while fixing the problem above, those two columns yield some NA values. This can be due to those rows having too many commas, forcing the values that should belong to one columns move to another column. We omit these outliers for a clean dataset by using the function na.omit.

Moreover, one of the famous authors that we know, J.K Rowling, has a fairly long name of “J.K Rowling-Mary GrandPré”. After changing the author’s name, we have the dataset available for further analysis.

goodreads[goodreads=='J.K. Rowling-Mary GrandPré'] <- 'J.K. Rowling'
head(goodreads)

for(i in c(4,8)) {
  goodreads[,i] <- as.numeric(goodreads[,i])
}

NAs introduced by coercionNAs introduced by coercion

names(goodreads)[names(goodreads) == "X..num_pages"] <- "num_pages"
goodreads <- na.omit(goodreads)
str(goodreads)  

'data.frame':   13714 obs. of  10 variables:
 $ bookID            : int  1 2 3 4 5 8 9 10 12 13 ...
 $ title             : chr  "Harry Potter and the Half-Blood Prince (Harry Potter  #6)" "Harry Potter and the Order of the Phoenix (Harry Potter  #5)" "Harry Potter and the Sorcerer's Stone (Harry Potter  #1)" "Harry Potter and the Chamber of Secrets (Harry Potter  #2)" ...
 $ authors           : chr  "J.K. Rowling" "J.K. Rowling" "J.K. Rowling" "J.K. Rowling" ...
 $ average_rating    : num  4.56 4.49 4.47 4.41 4.55 4.78 3.69 4.73 4.38 4.38 ...
 $ isbn              : chr  "0439785960" "0439358078" "0439554934" "0439554896" ...
 $ isbn13            : chr  "9780439785969" "9780439358071" "9780439554930" "9780439554893" ...
 $ language_code     : chr  "eng" "eng" "eng" "eng" ...
 $ num_pages         : num  652 870 320 352 435 ...
 $ ratings_count     : int  1944099 1996446 5629932 6267 2149872 38872 18 27410 3602 240189 ...
 $ text_reviews_count: int  26249 27613 70390 272 33964 154 1 820 258 3954 ...
 - attr(*, "na.action")= 'omit' Named int  4011 4012 5688 5689 7057 7058 10603 10604 10671 10672
  ..- attr(*, "names")= chr  "4011" "4012" "5688" "5689" ...

print(paste('The final goodreads book dataset has',nrow(goodreads),'rows and',ncol(goodreads),'columns.'))

[1] "The final goodreads book dataset has 13714 rows and 10 columns."

After fixing all the errors appeared in the dataset, we start to define the variables’ names. Their description is as follow:

• bookID: the unique ID for each book/series
• title: the title of each book/series
• authors: the author(s) of the particular book/series
• average_rating: the average ratings of the particular book/series, given by Goodreads’ users
• ISBN: old ISBN number of 10 digits, including information about a book/series, for example name, author(s), publisher, genre, etc.
• ISBN13: new ISBN number of 13 digits, introduced in 2007
• Language_code: 3-character language code, which indicates the language in which the book is written
• Num_page: number of pages of each book/series
• Ratings_count: number of ratings given by users for each book/series
• Text_reviews_count: number of text reviews (apart from point reviews of scale 5) given by users for each book/series

Missingness of data

We investigate whether there is any data cell missing with the vis_miss plot. Since we do not plot any significant in the plot, and it shows that less than 0.1% of the data is missing, we use gg_miss_var to get the exact number of data cells missing in each column. The result is, surprisingly, there is no data misising.

vis_miss(goodreads)

gg_miss_var(goodreads)

Exploratory Data Analysis (EDA) of the dataset Goodreads

In this part, we will answer some of the interesting exploratory questions by visualizing data.

Correlation between numerical columns

First, we will see the correlation between numerical columns of this dataset by using Pearson’s R, Kendall rank and Spearman correlation coefficient.

• Pearson’s R correlation coefficient is used to examine the strength and direction of the linear relationship between two continuous variables.
• Kendall rank correlation coefficient (or Kendall tau’s coefficient) is a statistic used to measure the ordinal association between two measured quantities.
• Spearman’s rank correlation coefficient (or Spearman’s rho) is a non-parametric measure of rank correlation (statistical dependence between the rankings of two variables). It assesses how well the relationship between two variables can be described using a monotonic function.

goodreads_corr <- subset(goodreads, select = c('average_rating','num_pages','ratings_count','text_reviews_count'))
goodreads_corr_pearson_round2 <- round(cor(goodreads_corr, method = "pearson", use = "pairwise.complete.obs"), 2)
data.frame(goodreads_corr_pearson_round2)

goodreads_corr_kendall_round2 <- round(cor(goodreads_corr, method = "kendall", use = "pairwise.complete.obs"), 2)
data.frame(goodreads_corr_kendall_round2)

goodreads_corr_spearman_round2 <- round(cor(goodreads_corr, method = "spearman", use = "pairwise.complete.obs"), 2)
data.frame(goodreads_corr_spearman_round2)

ggcorrplot(goodreads_corr_pearson_round2, hc.order = TRUE,
   lab = TRUE,
   title = "Pearson's R correlation matrix for Goodreads numerical variables",
   outline.col = "white",
   ggtheme = ggplot2::theme_gray,
   colors = c("#6D9EC1", "white", "#E46726"))

ggcorrplot(goodreads_corr_kendall_round2, hc.order = TRUE,
   lab = TRUE,
   title = "Kendall correlation matrix for Goodreads numerical variables",
   outline.col = "white",
   ggtheme = ggplot2::theme_gray,
   colors = c("#6D9EC1", "white", "#E46726"))

ggcorrplot(goodreads_corr_spearman_round2, hc.order = TRUE,
   lab = TRUE,
   title = "Spearman correlation matrix for Goodreads numerical variables",
   outline.col = "white",
   ggtheme = ggplot2::theme_gray,
   colors = c("#6D9EC1", "white", "#E46726"))

From the three correlation matrices above, we can see that there is only one pair of numerical variables that are strongly correlated to each other, which is ratings_count and text_reviews_count. All of the correlation coefficients of these two variables, regardless of the method used, are always greater than 0.8, which can be considered to be a strong correlation. The remaining pairs of variables show little positive correlation, with all the remaining coefficients of less than 0.2, some are even less than 0.05.

This shows an intuitive result that the more readers rate a book on a scale of 0 to 5, the more readers will leave a written text review for the same book.

Histogram plot of numerical variables

First, we will plot the histograms of the numerical/integer variables, including average_rating, num_pages, ratings_count and text_reviews_count.

ggplot(data = goodreads, aes(x = goodreads$average_rating)) + 
    geom_histogram(aes(y = ..density..),binwidth = 0.01, color='orange', fill='orange') +
    geom_density(colour="blue", lwd = 0.5, alpha=0.5) + 
    labs(title="Histogram for books' average ratings", x = "Average ratings", y = "Number of books") + 
    geom_vline(data = goodreads, xintercept = mean(goodreads$average_rating, na.rm = TRUE), color = "red", linetype = "dashed", size = 0.5)

ggplot(data = goodreads, aes(x = goodreads$num_pages)) + 
    geom_histogram(aes(y = ..density..),binwidth = 10, color='orange', fill='orange') +
    geom_density(colour="blue", lwd = 0.5, alpha=0.5) + 
    labs(title="Histogram for books' number of pages", x = "Number of pages", y = "Number of books") + 
    geom_vline(data = goodreads, xintercept = mean(goodreads$average_rating, na.rm = TRUE), color = "red", linetype = "dashed", size = 0.5)

ggplot(data = goodreads, aes(x = goodreads$ratings_count)) + 
    geom_histogram(aes(y = ..density..),binwidth = 100, color='orange', fill='orange') +
    geom_density(colour="blue", lwd = 0.5, alpha=0.5) + 
    labs(title="Histogram for ratings count of each book/series", x = "Ratings count", y = "Number of books") + 
    geom_vline(data = goodreads, xintercept = mean(goodreads$average_rating, na.rm = TRUE), color = "red", linetype = "dashed", size = 0.5)

ggplot(data = goodreads, aes(x = goodreads$text_reviews_count)) + 
    geom_histogram(aes(y = ..density..),binwidth = 50 , color='orange', fill='orange') +
    geom_density(colour="blue", lwd = 0.5, alpha=0.5) + 
    labs(title="Histogram for text reviews count for each book/series", x = "Text reviews count", y = "Number of books") + 
    geom_vline(data = goodreads, xintercept = mean(goodreads$average_rating, na.rm = TRUE), color = "red", linetype = "dashed", size = 0.5)

The histogram of average_ratings shows that most of the ratings lie between 3 and 5. The distribution of ratings between 3 and 5 seems to be familiar to a normal distribution (i.e most of the ratings focus around 4, and the density decreases upon reaching the edges). Later, we will investigate this distribution and test how close is it to the normal distribution.

With the histogram of num_pages, most of the books have less than 1000 pages, with many of them having 250-500 pages. Later, we will also zoom into this part of the histogram to have a closer look of the distribution.

The histograms of ratings_count and text_reviews_count is so much heavily skewed to the left that we can only see a straight small bar at the position 0, and the density curve decreases to 0 shortly. As such, to further investigate the distribution of these variables, we have to later take a closer look at the positions around 0.

Basic statistics of numerical variables

First, as R does not have a built in function to calculate the mode of a dataset, we proceed to define that function ourselves as follow.

# Define the function getmode in R to get the mode of data (appearance with highest frequency)
getmode <- function(v) {
   uniqv <- unique(v)
   uniqv[which.max(tabulate(match(v, uniqv)))]
}

Now, we will generate a table of basic properties of each variable.

basic_statistics_goodreads <- data.frame(matrix(NA, ncol = 13, nrow = 10))
names(basic_statistics_goodreads) <- c('Variable',
                             'Minimum score',
                             '25th percentile',
                             'Median',
                             'Mean score',
                             '75th percentile',
                             'Maximum score',
                             'Value with highest frequency (mode)',
                             'Frequency of mode',
                             'Standard Deviation',
                             'SE Mean',
                             'Skewness',
                             'Kurtosis')
for(i in c(4,8:10)) {
  basic_statistics_goodreads[i,1] <- colnames(goodreads)[i]
  basic_statistics_goodreads[i,2] <- min(goodreads[,i], na.rm = TRUE)
  basic_statistics_goodreads[i,3] <- quantile(goodreads[,i], 0.25, na.rm = TRUE)
  basic_statistics_goodreads[i,4] <- quantile(goodreads[,i], 0.5, na.rm = TRUE)
  basic_statistics_goodreads[i,5] <- mean(goodreads[,i], na.rm = TRUE)
  basic_statistics_goodreads[i,6] <- quantile(goodreads[,i], 0.75, na.rm = TRUE)
  basic_statistics_goodreads[i,7] <- max(goodreads[,i], na.rm = TRUE)
  basic_statistics_goodreads[i,8] <- getmode(na.omit(goodreads[,i]))
  basic_statistics_goodreads[i,9] <- sum(goodreads[,i] == basic_statistics_goodreads[i,8])
  basic_statistics_goodreads[i,10] <- sd(goodreads[,i], na.rm = TRUE)
  basic_statistics_goodreads[i,11] <- std.error(goodreads[,i], na.rm = TRUE)
  basic_statistics_goodreads[i,12] <- skewness(goodreads[,i], na.rm = TRUE)
  basic_statistics_goodreads[i,13] <- kurtosis(goodreads[,i], na.rm = TRUE)
}
basic_statistics_goodreads <- na.omit(basic_statistics_goodreads)
View(basic_statistics_goodreads)

First of all, we see an interesting fact that the mode of text_reviews_count is 0 (more than 900 books receive 0 text reviews), which means that books having 0 text reviews occur the most frequently in the dataset. Moreover, the mode of average_ratings is also exactly 4.0, which shows that readers are really generous with the books they are reading.

Among the 4 variables, there is only one variable (average_ratings) that has a negative skewness. This is due to the fact that the histogram of this variable skews heavily to the right, with most of the observations lie between 3 and 5. The three remaining variables has a significantly positive skewness, which means that these variables skew heavily to the left. This can be verified by looking at the histograms of these variables.

We also note that the 75th percentile of each variable, apart from average_ratings, is relatively very small comparing to the maximum, which can also explain the positive skewness of those variables.

All 4 variables have a positive kurtosis, which shows that the distribution of these variables have heavier tails than the normal distribution with the same mean and standard deviation.

From the properties witnessed above, to better understand the distribution of variables within specific ranges, we again generate histograms for two variables, ratings_count and text_reviews_count. This time, we will set the limit of the x-axis to be 100 for num_pages, 5000 for ratings_count and 500 for text_reviews_count.

ggplot(data = goodreads, aes(x = goodreads$num_pages)) + 
    geom_histogram(aes(y = ..density..),binwidth = 20, color='orange', fill='orange') +
    xlim(c(0,1000)) + 
    geom_density(colour="blue", lwd = 0.5, alpha=0.5) + 
    labs(title="Histogram for books' number of pages", x = "Number of pages", y = "Number of books") + 
    geom_vline(data = goodreads, xintercept = mean(goodreads$average_rating, na.rm = TRUE), color = "red", linetype = "dashed", size = 0.5)

ggplot(data = goodreads, aes(x = goodreads$ratings_count)) + 
    geom_histogram(aes(y = ..density..),binwidth = 10, color='orange', fill='orange') +
    xlim(c(0,5000)) + 
    geom_density(colour="blue", lwd = 0.5, alpha=0.5) + 
    labs(title="Histogram for ratings count of each book/series", x = "Ratings count", y = "Number of books") + 
    geom_vline(data = goodreads, xintercept = mean(goodreads$average_rating, na.rm = TRUE), color = "red", linetype = "dashed", size = 0.5)

ggplot(data = goodreads, aes(x = goodreads$text_reviews_count)) + 
    geom_histogram(aes(y = ..density..),binwidth = 1 , color='orange', fill='orange') +
    xlim(c(0,500)) + 
    geom_density(colour="blue", lwd = 0.5, alpha=0.5) + 
    labs(title="Histogram for text reviews count for each book/series", x = "Text reviews count", y = "Number of books") + 
    geom_vline(data = goodreads, xintercept = mean(goodreads$average_rating, na.rm = TRUE), color = "red", linetype = "dashed", size = 0.5)

After plotting the histograms of three variables again, we see that the ratings_count and text_reviews_count still skew heavily to the left. The mode of these two variables 0 and 3, respectively, which also add to the fact that most books receive very few ratings/reviews.

Considering number of pages, most books have the number of pages of around 175 - 325. Then, the number of book inversely proportional to the number of pages (i.e the number of books is decreasing with respect to their length).

Scatterplot of pairs of numerical variables in the dataset

ggpairs(with(goodreads, data.frame(average_rating, num_pages, ratings_count, text_reviews_count)))

From this scatterplot matrix, we can see that, apart from the pairs ratings_count and text_reviews_count, the remaining shows little to no correlation. All of the scatterplot shows a nearly vertical trend line (i.e there is no linear correlation between the two variables plotted).

Rating distribustion of books in the dataset

First, we divide the ratings into five different ranges:

• Between 0 and 1
• Between 1 and 2
• Between 2 and 3
• Between 3 and 4
• Between 4 and 5

We calculate the number of books within each range, along with their percentages.

rating_distribution_goodreads <- data.frame(matrix(0, ncol = 3, nrow = 5))
names(rating_distribution_goodreads) <- c('Range',
                             'number_of_books',
                             'percentage')
for(i in c(1:5)) {
  rating_distribution_goodreads[i,1] <- paste('Between',i-1,'and',i)
}
for(i in c(1:nrow(goodreads))) {
  if (goodreads[i,4] >= 0 & goodreads[i,4] < 1) {
    rating_distribution_goodreads[1,2] = rating_distribution_goodreads[1,2] + 1 
  }
  if (goodreads[i,4] >= 1 & goodreads[i,4] < 2) {
    rating_distribution_goodreads[2,2] = rating_distribution_goodreads[2,2] + 1 
    }
  if (goodreads[i,4] >= 2 & goodreads[i,4] < 3) {
    rating_distribution_goodreads[3,2] = rating_distribution_goodreads[3,2] + 1 
    }
  if (goodreads[i,4] >= 3 & goodreads[i,4] < 4) {
    rating_distribution_goodreads[4,2] = rating_distribution_goodreads[4,2] + 1 
    }
  if (goodreads[i,4] >= 4 & goodreads[i,4] < 5) {
    rating_distribution_goodreads[5,2] = rating_distribution_goodreads[5,2] + 1 
  }
}
rating_distribution_goodreads$percentage <- round((rating_distribution_goodreads$number_of_books)/nrow(goodreads)*100,digits = 2)
rating_distribution_goodreads

Then, we will plot two different pie charts, a traditional one and a square pie chart, to observe the distribution of books within ranges.

ggplot(data=rating_distribution_goodreads, aes(x="", y=percentage,
       fill = factor(rating_distribution_goodreads[,1]), )) +
       geom_bar(width = 1, stat = "identity") + 
       ggtitle("Pie chart of percentage of Books with respect to ratings count") + 
       coord_polar(theta="y", start = 0) +
       labs(fill = factor(rating_distribution_goodreads[,1]))

rating_distribution_number <- c(`Between 0 and 1`= 0.25, `Between 1 and 2`= 0.02, `Between 2 and 3`= 0.53, `Between 3 and 4`= 55.32, `Between 4 and 5`= 43.69)
waffle(rating_distribution_number, rows=6, 
       colors=brewer.pal(5,"Set1"),
       title="Percentage of Books with respect to ratings count")

We can see that, more than half of the books are rated between 3 and 4 points, while most the remaining books are ranked between 4 and 5. There are a total of 13577 books rated between 3 and 5, which counts up to 99% of the books in the dataset.

Books with the most occurences in the dataset

book_frequency <- data.frame(table(goodreads$title), stringsAsFactors = FALSE)
names(book_frequency) <- c('title','frequency')
book_frequency <- book_frequency[order(-book_frequency$frequency),] 
head(book_frequency)

book_frequency_top20 <- book_frequency[c(1:20),]
ggplot(data=book_frequency_top20, aes(x=reorder(title, frequency),y=frequency, fill = factor(c(1:nrow(book_frequency_top20))))) +
  theme(legend.position = "none") + 
  geom_bar(stat="identity") +
  coord_flip() +
  ggtitle('20 books with highest frequency') + 
  labs(x = "Title", y = "Frequency") 

Above, we have plotted the 20 books with the most appearances in the dataset. We can see that “One Hundred Year of Solitude” and “Salem’s Lot” are the two books appearing the most, with 11 times for each book.

These books are published again and again, each time with another publisher. This shows that these are still in great demand, despite the flow of time.

Languages written in books/series

book_language <- data.frame(table(goodreads$language_code), stringsAsFactors = FALSE)
names(book_language) <- c('language','frequency')
book_language <- book_language[order(-book_language$frequency),]
book_language$percentage <- round(book_language$frequency/nrow(goodreads)*100, digits = 2)
head(book_language)

ggplot(book_language, aes(x=reorder(language,-frequency),y=frequency, fill = factor(c(1:nrow(book_language))))) +
  theme(legend.position = "none") + 
  geom_bar(stat="identity") +
  ggtitle('Languages written in books/series') + 
  labs(x = "Language", y = "Total number of books") +
  geom_text(aes(label=frequency), vjust=-0.5)

We can easily see that most of the books in this dataset is written in English. Two most used languages are English and English-US, while within 6 most commonly used languages, English and its different versions (eng-US and eng-GB) counts up to half of them. Combining all the different versions of this language, English is written in 12634 books, which is 92,12% of all the books. Apart from English, Spanish, the third commonly used language, only has 419 books, which is 3.1% of the books in this dataset.

10 Books with the highest average rating

book_average_rating <- goodreads[order(-goodreads$average_rating),]
book_average_rating_top10 <- book_average_rating[c(1:10),]
book_average_rating_top10

ggplot(data=book_average_rating_top10, aes(x=reorder(title, average_rating),y=average_rating, fill = factor(c(1:nrow(book_average_rating_top10))))) +
  theme(legend.position = "none") + 
  geom_bar(stat="identity") +
  coord_flip() +
  scale_fill_brewer(palette="Paired") + 
  ggtitle('Books/series with highest average rating') + 
  labs(x = "Book/series", y = "Average rating") +
  geom_text(
    aes(label = average_rating), colour = "black",
    hjust = -0.1, size = 2,
    position = position_dodge(width = 1),
    inherit.aes = TRUE
  )

author_highest_book_average_rating <- data.frame(table(book_average_rating_top10$authors))
names(author_highest_book_average_rating) <- c('author', 'number_of_books')
author_highest_book_average_rating <- author_highest_book_average_rating[order(-author_highest_book_average_rating$number_of_books),]
author_highest_book_average_rating

Surprisingly, all 10 of the highest rating books have the maximum rating, 5. As a result, we will investigate all the books that have the rating of 5 in this dataset.

books_rating_5 <- goodreads[goodreads$average_rating == 5,]
books_rating_5 <- books_rating_5[order(-books_rating_5$ratings_count),]
books_rating_5 

print(paste('There are',nrow(books_rating_5),'books that receive an average rating of 5.'))

[1] "There are 28 books that receive an average rating of 5."

We can easily see that all of the books with average ratings of 5 receive very few ratings. The maximum ratings count among these books are 5. Specifically, there are 4 books that receive 0 ratings, and still receive an average rating of 5, which is not supposed to happen. Moreover, 23/28 books in the list receive 0 text reviews. We can not consider these books to be the ones with highest average rating, as the ratings count is too low.

Instead, we propose the following method: We only consider the average rating of books with more than 1000 ratings.

book_1000_ratings_count <- goodreads[goodreads$ratings_count >= 1000,]
book_1000_ratings_count <- book_1000_ratings_count[order(-book_1000_ratings_count$average_rating),]
book_1000_ratings_count

book_1000_ratings_count_top10 <- book_1000_ratings_count[c(1:10),]
book_1000_ratings_count_top10

ggplot(data=book_1000_ratings_count_top10, aes(x=reorder(title, average_rating),y=average_rating, fill = factor(c(1:nrow(book_1000_ratings_count_top10))))) +
  theme(legend.position = "none") + 
  geom_bar(stat="identity") +
  coord_flip() +
  scale_fill_brewer(palette="Paired") + 
  ggtitle('Books/series with highest average rating (ratings count >= 1000)') + 
  labs(x = "Book/series", y = "Average rating") +
  geom_text(
    aes(label = average_rating), colour = "black",
    hjust = -0.1, size = 2,
    position = position_dodge(width = 1),
    inherit.aes = TRUE
  )

author_book_1000_ratings_count <- data.frame(table(book_1000_ratings_count_top10$authors))
names(author_book_1000_ratings_count) <- c('author', 'number_of_books')
author_book_1000_ratings_count <- author_book_1000_ratings_count[order(-author_book_1000_ratings_count$number_of_books),]
author_book_1000_ratings_count

We can see that, there is only 6059 books that have the ratings count of more than or equal to 1000, which accounts for 44.18% of all the books in the dataset.

Within these 6059 books, the highest has the average rating of 4.82, while the lowest rating among the top 10 books is 4.7.

Among those 10 books, half of them are written by Bill Watterson, two are from J.K. Rowling, one from Patrick O’Brian, one from Joyce Meyer and one from anonymous. We can easily see that Bill Watterson has written books that receive very high ratings from a large number of readers. Two among the top 10 from J.K Rowling are both sets of Harry Potter (one set of #1-5 and one set of #1-6).

Another interesting fact is that all books in top 10 are written in English, with one specifically written in en-US.

10 longest books in the dataset

book_num_pages <- goodreads[order(-goodreads$num_pages),]
book_num_pages_top10 <- book_num_pages[c(1:10),]
book_num_pages_top10

ggplot(data=book_num_pages_top10, aes(x=reorder(title, num_pages),y=num_pages, fill = factor(c(1:nrow(book_num_pages_top10))))) +
  theme(legend.position = "none") + 
  geom_bar(stat="identity") +
  coord_flip() +
  scale_fill_brewer(palette="Paired") + 
  ggtitle('Books/series with highest number of pages') + 
  labs(x = "Book/series", y = "Number of pages") +
  geom_text(
    aes(label = num_pages), colour = "black",
    hjust = -0.1, size = 2,
    position = position_dodge(width = 1),
    inherit.aes = TRUE
  )

author_book_num_pages <- data.frame(table(book_num_pages_top10$authors))
names(author_book_num_pages) <- c('author', 'number_of_books')
author_book_num_pages <- author_book_num_pages[order(-author_book_num_pages$number_of_books),]
author_book_num_pages

We can easily see that, all of the 10 longest in this dataset are all series. Among them, “The Complet Aubrey/Maturin Novels” stands out from the crowd, with its length of nearly 3 times the length of the 10th longest series, “Civil War: a Narrative”. Moreover, “The Norton Anthology of English Literature” has 3 of their volumns presenting in the top 10 (Vols A-C, Vol 2 and Vol 1). However, their appearances are not consistent: Vols A-C and Vol 1 have the same name, but their number of pages are different.

10 Books with the highest number of ratings count

book_ratings_count <- goodreads[order(-goodreads$ratings_count),]
book_ratings_count_top10 <- book_ratings_count[c(1:10),]
book_ratings_count_top10

ggplot(data=book_ratings_count_top10, aes(x=reorder(title, ratings_count),y=ratings_count, fill = factor(c(1:nrow(book_ratings_count_top10))))) +
  theme(legend.position = "none") + 
  geom_bar(stat="identity") +
  coord_flip() +
  scale_fill_brewer(palette="Paired") + 
  ggtitle('Books/series with highest number of ratings count') + 
  labs(x = "Book/series", y = "Number of ratings") +
  geom_text(
    aes(label = ratings_count), colour = "black",
    hjust = -0.1, size = 2,
    position = position_dodge(width = 1),
    inherit.aes = TRUE
  )

author_most_book_ratings_count <- data.frame(table(book_ratings_count_top10$authors))
names(author_most_book_ratings_count) <- c('author', 'number_of_books')
author_most_book_ratings_count <- author_most_book_ratings_count[order(-author_most_book_ratings_count$number_of_books),]
author_most_book_ratings_count

There is a huge gap between the first two most rated books and the remainings. Specifically speaking, there is a 2 million rating gap between Twilight #1 - the second most rated book and “The Hobbit or Three and Back Again” - the third most rated book. Even between the first position, “Harry Potter and the Sorcerer’s Stone” and the second one, there is also a significant gap of about 1.3 million ratings. However, between the third and the 10th position, the gap is just about 350.000 ratings.

About the authors’ list of top 10 books, we can easily see that J.K Rowling and her Harry Potter series are dominating the ratings count, with 4 of her books in the series making their ways to the top 10. - Harry Potter and the Sorcerer’s Stone (Harry Potter #1) at first place - Harry Potter and the Prisoner of Azkaban (Harry Potter #3) at sixth place - Harry Potter and the Chamber of Secrets (Harry Potter #2) at seventh place - Harry Potter and the Order of the Phoenix (Harry Potter #5) at tenth place

Large number of ratings count, alongside with high average rating (all over 4.4), these definitely shows how successful this series have been.

10 Books with the highest number of ratings count

book_text_reviews_count <- goodreads[order(-goodreads$text_reviews_count),]
book_text_reviews_count_top10 <- book_text_reviews_count[c(1:10),]
book_text_reviews_count_top10

ggplot(data=book_text_reviews_count_top10, aes(x=reorder(title, text_reviews_count),y=text_reviews_count, fill = factor(c(1:nrow(book_text_reviews_count_top10))))) +
  theme(legend.position = "none") + 
  geom_bar(stat="identity") +
  coord_flip() +
  scale_fill_brewer(palette="Paired") + 
  ggtitle('Books/series with highest number of text reviews') + 
  labs(x = "Book/series", y = "Number of text reviews") +
  geom_text(
    aes(label = text_reviews_count), colour = "black",
    hjust = -0.1, size = 2,
    position = position_dodge(width = 1),
    inherit.aes = TRUE
  )

author_most_book_text_reviews_count <- data.frame(table(book_text_reviews_count_top10$authors))
names(author_most_book_text_reviews_count) <- c('author', 'number_of_books')
author_most_book_text_reviews_count <- author_most_book_text_reviews_count[order(-author_most_book_text_reviews_count$number_of_books),]
author_most_book_text_reviews_count

Once again, there is a significant gap between the first four positions; after that, the gap is getting much narrower.

Moreover, this time, the author’s list of these books are more widely spread, with 10 different authors, each having one of their books in the top 10 list.

We have seen some familiar names that appear in both top 10 of ratings count and text reviews count. The question is, how many of them are there, and what is the difference between their performance in these two criteria?

Common books of the two top 10 of ratings count and text reviews count

common_top10 <- intersect(book_text_reviews_count_top10, book_ratings_count_top10)  
common_top10

Although there is a very strong positive correlation between the two variables, there are only two books that belong to both top 10 of these two criteria. They are:

• Twilight (Twilight #1)
• Harry Potter and the Sorcerer’s Stone (Harry Potter #1)

These two books are performing extremely well in both criteria, and so far the leading books in this dataset, considering the number of ratings/reviews:

• With Twilight (Twilight #1): Top 2 in ratings count, Top 1 in text reviews count
• With Harry Potter and the Sorcerer’s Stone (Harry Potter #1): Top 1 in ratings count, Top 3 in text reviews count.

author_most_books <- data.frame(table(goodreads$authors), stringsAsFactors = FALSE)
names(author_most_books) <- c('author','total_number_of_books')
author_most_books <- author_most_books[order(-author_most_books$total_number_of_books),] 
head(author_most_books)

author_most_books_top10 <- author_most_books[c(1:10),]
ggplot(data=author_most_books_top10, aes(x=reorder(author, total_number_of_books),y=total_number_of_books, fill = factor(c(1:nrow(author_most_books_top10))))) +
  theme(legend.position = "none") + 
  geom_bar(stat="identity") +
  coord_flip() +
  scale_fill_brewer(palette="Set3") + 
  ggtitle('Author with greatest number of books appeared') + 
  labs(x = "Author", y = "Total number of books") +
  geom_text(
    aes(label = total_number_of_books), colour = "black",
    hjust = -0.5, size = 3,
    position = position_dodge(width = 1),
    inherit.aes = TRUE
  )

author_most_books_top10

Once again, the first two authors creates a huge gap with the remaining. Both Agatha Christie and Stephen King have more than 65 books, which is at least 18 books more than the third author, Orson Scott Card.

What are the authors with the greatest number of highly-ranked books?

The definition of “highly-ranked books” in this context is relative. As such, we consider some of the following cases. We define a book to be highly-ranked if its average ranking is greater than:

• 4.0
• 4.3
• 4.5

print(paste('There are',sum(goodreads$average_rating >= 4.0),'books that have a rating of greater than or equal to 4.0, which accounts for',round(sum(goodreads$average_rating >= 4.0)/nrow(goodreads)*100, digits = 2),'percent of the books in this dataset.'))

[1] "There are 6019 books that have a rating of greater than or equal to 4.0, which accounts for 43.89 percent of the books in this dataset."

print(paste('There are',sum(goodreads$average_rating >= 4.3),'books that have a rating of greater than or equal to 4.3, which accounts for',round(sum(goodreads$average_rating >= 4.3)/nrow(goodreads)*100, digits = 2),'percent of the books in this dataset.'))

[1] "There are 1301 books that have a rating of greater than or equal to 4.3, which accounts for 9.49 percent of the books in this dataset."

print(paste('There are',sum(goodreads$average_rating >= 4.5),'books that have a rating of greater than or equal to 4.5, which accounts for',round(sum(goodreads$average_rating >= 4.5)/nrow(goodreads)*100, digits = 2),'percent of the books in this dataset.'))

[1] "There are 271 books that have a rating of greater than or equal to 4.5, which accounts for 1.98 percent of the books in this dataset."

author_most_book_high_ratings <- goodreads[goodreads$average_rating >= 4.0,]
author_most_books_high_ratings <- data.frame(table(author_most_book_high_ratings$authors), stringsAsFactors = FALSE)
names(author_most_books_high_ratings) <- c('author','number_of_books_>=_4.0')
author_most_books_high_ratings <- author_most_books_high_ratings[order(-author_most_books_high_ratings$`number_of_books_>=_4.0`),] 
author_most_book_high_ratings_2 <- goodreads[goodreads$average_rating >= 4.3,]
author_most_books_high_ratings_2 <- data.frame(table(author_most_book_high_ratings_2$authors), stringsAsFactors = FALSE)
names(author_most_books_high_ratings_2) <- c('author','number_of_books_>=_4.3')
author_most_books_high_ratings_2 <- author_most_books_high_ratings_2[order(-author_most_books_high_ratings_2$`number_of_books_>=_4.3`),] 
author_most_book_high_ratings_3 <- goodreads[goodreads$average_rating >= 4.5,]
author_most_books_high_ratings_3 <- data.frame(table(author_most_book_high_ratings_3$authors), stringsAsFactors = FALSE)
names(author_most_books_high_ratings_3) <- c('author','number_of_books_>=_4.5')
author_most_books_high_ratings_3 <- author_most_books_high_ratings_3[order(-author_most_books_high_ratings_3$`number_of_books_>=_4.5`),] 
author_most_books_high_ratings_top10 <- author_most_books_high_ratings[c(1:10),]
ggplot(data=author_most_books_high_ratings_top10, aes(x=reorder(author, `number_of_books_>=_4.0`),y=`number_of_books_>=_4.0`, fill = factor(c(1:nrow(author_most_books_high_ratings_top10))))) +
  theme(legend.position = "none") + 
  geom_bar(stat="identity") +
  coord_flip() +
  scale_fill_brewer(palette="Set3") + 
  ggtitle('Author with greatest number of books ranked >= 4.0') + 
  labs(x = "Author", y = "Total number of highly books ranked >= 4.0") +
  geom_text(
    aes(label = `number_of_books_>=_4.0`), colour = "black",
    hjust = -0.5, size = 3,
    position = position_dodge(width = 1),
    inherit.aes = TRUE
  )

author_most_books_high_ratings_top10_2 <- author_most_books_high_ratings_2[c(1:10),]
ggplot(data=author_most_books_high_ratings_top10_2, aes(x=reorder(author, `number_of_books_>=_4.3`),y=`number_of_books_>=_4.3`, fill = factor(c(1:nrow(author_most_books_high_ratings_top10_2))))) +
  theme(legend.position = "none") + 
  geom_bar(stat="identity") +
  coord_flip() +
  scale_fill_brewer(palette="Set3") + 
  ggtitle('Author with greatest number of books ranked >= 4.3') + 
  labs(x = "Author", y = "Total number of books ranked >= 4.3") +
  geom_text(
    aes(label = `number_of_books_>=_4.3`), colour = "black",
    hjust = -0.5, size = 3,
    position = position_dodge(width = 1),
    inherit.aes = TRUE
  )

author_most_books_high_ratings_top10_3 <- author_most_books_high_ratings_3[c(1:10),]
ggplot(data=author_most_books_high_ratings_top10_3, aes(x=reorder(author, `number_of_books_>=_4.5`),y=`number_of_books_>=_4.5`, fill = factor(c(1:nrow(author_most_books_high_ratings_top10_3))))) +
  theme(legend.position = "none") + 
  geom_bar(stat="identity") +
  coord_flip() +
  scale_fill_brewer(palette="Set3") + 
  ggtitle('Author with greatest number of books ranked >= 4.5') + 
  labs(x = "Author", y = "Total number of books ranked >= 4.5") +
  geom_text(
    aes(label = `number_of_books_>=_4.5`), colour = "black",
    hjust = -0.5, size = 3,
    position = position_dodge(width = 1),
    inherit.aes = TRUE
  )

list_authors_top10s <- Reduce(function(x,y) merge(x,y,by="author",all=TRUE) ,list(author_most_books_high_ratings_top10,author_most_books_high_ratings_top10_2,author_most_books_high_ratings_top10_3))
list_authors_top10s

From the information gathered above, we can subjectively define the quality of a book based on its rating as follow:

• Between 4.0 and 4.3: Average/Fair
• Between 4.3 and 4.5: Good
• Above 4.5: Excellent

There are a total of 21 authors that appear in the three top 10 of authors who has the greatest number of books (with 3 different cutoffs). From the table above, we have the following observations:

• There are some authors who write a lot of above average/fair books, but only some or none of them are good or above. This means that most of their books are ranked between 4.0 and 4.3. Some authors in this group are Janet Evanovich, Mercedes Lackey, Stephen King, Rumiko Takahashi, Terry Pratchett, Agatha Christie. They are in top 10 with the 4.0 cutoff, but can not make it to top 10 with higher cutoffs.
  • For example, Rumiko Takahashi has 44 books ranked 4.0 or above, but he has fewer books ranked over 4.3 than Hiromu Arakawa (8 books).
• On the other hand, there are some authors who write fewer books, but most of their books are ranked fairly high. The two most noble names for this group are ill Watterson and J.K.Rowling.
  • Bill Watterson are not in top 10 with books over 4.0, but he has 16 books with ratings greater than or equal to 4.3, 15 of which has ratings greater than or equal to 4.5
  • J.K. Rowling has 31 books above average, all of them having ratings greater than or equal to 4.3, and 10 of them has ratings greater than or equal to 4.5
  • Hiromu Arakawa - Akira Watanabe has only 12 books whose ratings are greater than or equal to 4.3, but 11 of them have ratings greater than or equal to 4.5
• There are some authors who write very few books (i.e does not appear in top 10 of cutoffs 4.0 and 4.3), but most/all of their books are very highly rated (greater than or equal to 4.5).
  • Hayao Miyazaki with 6 books rated greater than or equal to 4.5
  • Hiromu Arakawa and Jane Austen with 4 books rated greater than or equal to 4.5

We can also see that, the distance between 4.0 and 4.3 is only 0.3, but the amount of books rated has decreased 5 times.

Applying K-means algorithm to cluster datasets

K-means clustering is the most commonly used unsupervised machine learning algorithm for partitioning a given data set into a set of k groups (i.e. k clusters), where k represents the number of groups pre-specified by the analyst.

First, we will build a model to cluster this dataset based on two variables: average_rating and ratings_count.

To determine the optimal number o clusters, we use the two following popular methods:

• Elbow method
• Sihouette method

goodreads_kmeans1 <- goodreads[,c('average_rating','ratings_count')]
set.seed(123)
fviz_nbclust(goodreads_kmeans1, kmeans, method = "wss")

fviz_nbclust(goodreads_kmeans1, kmeans, method = "silhouette")

From both of the two approaches above, the optimal number of clusters seem to be at 5. As a result, we will perform an analysis and extract the results with 5 clusters.

set.seed(123)
final_kmeans1 <- kmeans(goodreads_kmeans1, 5, nstart = 25)
print(final_kmeans1)

K-means clustering with 5 clusters of sizes 23, 13257, 357, 75, 2

Cluster means:
  average_rating ratings_count
1       4.100000   1760033.261
2       3.927476      5645.072
3       4.015490    191223.644
4       4.027733    667374.147
5       4.030000   4998636.500

Clustering vector:
   1    2    3    4    5    6    7    8    9   10   11   12   13   14   15   16   17   18   19   20   21   22   23   24   25   26   27   28   29 
   1    1    5    2    1    2    2    2    2    3    2    2    2    3    2    2    2    2    2    2    2    2    2    2    2    1    2    2    2 
  30   31   32   33   34   35   36   37   38   39   40   41   42   43   44   45   46   47   48   49   50   51   52   53   54   55   56   57   58 
   2    2    2    3    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2 
  59   60   61   62   63   64   65   66   67   68   69   70   71   72   73   74   75   76   77   78   79   80   81   82   83   84   85   86   87 
   2    2    2    2    2    2    3    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2 
  88   89   90   91   92   93   94   95   96   97   98   99  100  101  102  103  104  105  106  107  108  109  110  111  112  113  114  115  116 
   2    2    2    2    2    2    2    2    3    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2 
 117  118  119  120  121  122  123  124  125  126  127  128  129  130  131  132  133  134  135  136  137  138  139  140  141  142  143  144  145 
   2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2 
 146  147  148  149  150  151  152  153  154  155  156  157  158  159  160  161  162  163  164  165  166  167  168  169  170  171  172  173  174 
   3    2    2    2    2    2    2    2    4    2    2    2    2    2    2    2    3    2    2    2    3    2    2    2    2    2    2    2    2 
 175  176  177  178  179  180  181  182  183  184  185  186  187  188  189  190  191  192  193  194  195  196  197  198  199  200  201  202  203 
   2    2    3    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2 
 204  205  206  207  208  209  210  211  212  213  214  215  216  217  218  219  220  221  222  223  224  225  226  227  228  229  230  231  232 
   2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2 
 233  234  235  236  237  238  239  240  241  242  243  244  245  246  247  248  249  250  251  252  253  254  255  256  257  258  259  260  261 
   2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    3    2    2    2    2    2    3    2    2    3    2    2 
 262  263  264  265  266  267  268  269  270  271  272  273  274  275  276  277  278  279  280  281  282  283  284  285  286  287  288  289  290 
   2    3    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2 
 291  292  293  294  295  296  297  298  299  300  301  302  303  304  305  306  307  308  309  310  311  312  313  314  315  316  317  318  319 
   2    2    2    2    2    2    2    2    2    2    2    2    2    2    3    2    2    2    2    2    2    2    2    1    2    2    2    3    2 
 320  321  322  323  324  325  326  327  328  329  330  331  332  333  334  335  336  337  338  339  340  341  342  343  344  345  346  347  348 
   2    2    2    2    2    1    2    2    3    2    3    1    2    2    2    2    2    2    2    2    2    1    2    2    1    2    2    2    4 
 349  350  351  352  353  354  355  356  357  358  359  360  361  362  363  364  365  366  367  368  369  370  371  372  373  374  375  376  377 
   2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    3    2    2    2    3    2    2    2    3    2    3    2    2    2 
 378  379  380  381  382  383  384  385  386  387  388  389  390  391  392  393  394  395  396  397  398  399  400  401  402  403  404  405  406 
   2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    4    2    2    2    2    2    2    2    2    3    2    2    3 
 407  408  409  410  411  412  413  414  415  416  417  418  419  420  421  422  423  424  425  426  427  428  429  430  431  432  433  434  435 
   2    2    2    2    2    2    2    3    2    2    2    2    2    3    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2 
 436  437  438  439  440  441  442  443  444  445  446  447  448  449  450  451  452  453  454  455  456  457  458  459  460  461  462  463  464 
   2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    3    2    2    2    2    2    2 
 465  466  467  468  469  470  471  472  473  474  475  476  477  478  479  480  481  482  483  484  485  486  487  488  489  490  491  492  493 
   4    2    2    2    2    2    2    2    2    2    2    2    2    4    2    2    2    2    2    2    2    2    3    3    2    2    2    2    2 
 494  495  496  497  498  499  500  501  502  503  504  505  506  507  508  509  510  511  512  513  514  515  516  517  518  519  520  521  522 
   2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2 
 523  524  525  526  527  528  529  530  531  532  533  534  535  536  537  538  539  540  541  542  543  544  545  546  547  548  549  550  551 
   2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    3    2    2 
 552  553  554  555  556  557  558  559  560  561  562  563  564  565  566  567  568  569  570  571  572  573  574  575  576  577  578  579  580 
   2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    4    4    2    3    2    3    2    2 
 581  582  583  584  585  586  587  588  589  590  591  592  593  594  595  596  597  598  599  600  601  602  603  604  605  606  607  608  609 
   2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2 
 610  611  612  613  614  615  616  617  618  619  620  621  622  623  624  625  626  627  628  629  630  631  632  633  634  635  636  637  638 
   2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    3    2    4    2    2 
 639  640  641  642  643  644  645  646  647  648  649  650  651  652  653  654  655  656  657  658  659  660  661  662  663  664  665  666  667 
   2    2    3    2    2    3    2    2    2    2    2    2    2    2    2    2    2    2    2    3    2    2    2    2    2    2    1    2    2 
 668  669  670  671  672  673  674  675  676  677  678  679  680  681  682  683  684  685  686  687  688  689  690  691  692  693  694  695  696 
   2    2    2    2    4    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2 
 697  698  699  700  701  702  703  704  705  706  707  708  709  710  711  712  713  714  715  716  717  718  719  720  721  722  723  724  725 
   2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2 
 726  727  728  729  730  731  732  733  734  735  736  737  738  739  740  741  742  743  744  745  746  747  748  749  750  751  752  753  754 
   2    2    2    2    2    2    2    2    2    2    2    2    2    3    2    2    2    2    2    2    2    2    2    2    2    2    2    2    3 
 755  756  757  758  759  760  761  762  763  764  765  766  767  768  769  770  771  772  773  774  775  776  777  778  779  780  781  782  783 
   2    4    2    2    2    2    3    2    2    2    2    2    2    2    4    3    2    3    2    2    2    2    2    2    2    2    2    2    2 
 784  785  786  787  788  789  790  791  792  793  794  795  796  797  798  799  800  801  802  803  804  805  806  807  808  809  810  811  812 
   2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2 
 813  814  815  816  817  818  819  820  821  822  823  824  825  826  827  828  829  830  831  832  833  834  835  836  837  838  839  840  841 
   2    2    2    2    2    2    2    2    2    2    2    3    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2 
 842  843  844  845  846  847  848  849  850  851  852  853  854  855  856  857  858  859  860  861  862  863  864  865  866  867  868  869  870 
   2    2    2    2    2    2    2    3    2    2    2    2    2    2    2    2    2    2    2    3    2    2    2    2    2    2    2    2    2 
 871  872  873  874  875  876  877  878  879  880  881  882  883  884  885  886  887  888  889  890  891  892  893  894  895  896  897  898  899 
   2    2    2    2    3    2    2    2    2    2    2    2    2    2    2    2    2    4    2    2    2    4    2    2    2    2    2    2    2 
 900  901  902  903  904  905  906  907  908  909  910  911  912  913  914  915  916  917  918  919  920  921  922  923  924  925  926  927  928 
   2    2    2    2    2    2    2    2    2    2    2    3    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2 
 929  930  931  932  933  934  935  936  937  938  939  940  941  942  943  944  945  946  947  948  949  950  951  952  953  954  955  956  957 
   2    3    2    2    2    2    2    2    2    3    2    2    2    2    3    2    2    2    2    2    2    2    2    2    2    2    2    2    2 
 958  959  960  961  962  963  964  965  966  967  968  969  970  971  972  973  974  975  976  977  978  979  980  981  982  983  984  985  986 
   2    2    2    2    3    2    2    2    2    2    2    2    2    2    4    2    2    2    2    2    2    2    2    2    3    2    2    2    2 
 987  988  989  990  991  992  993  994  995  996  997  998  999 1000 
   2    2    2    2    2    2    2    2    2    2    2    2    2    2 
 [ reached getOption("max.print") -- omitted 12714 entries ]

Within cluster sum of squares by cluster:
[1] 2.834358e+12 2.385364e+12 2.431455e+12 2.747742e+12 7.970680e+11
 (between_SS / total_SS =  93.6 %)

Available components:

[1] "cluster"      "centers"      "totss"        "withinss"     "tot.withinss" "betweenss"    "size"         "iter"         "ifault"      

str(final_kmeans1)

List of 9
 $ cluster     : Named int [1:13714] 1 1 5 2 1 2 2 2 2 3 ...
  ..- attr(*, "names")= chr [1:13714] "1" "2" "3" "4" ...
 $ centers     : num [1:5, 1:2] 4.1 3.93 4.02 4.03 4.03 ...
  ..- attr(*, "dimnames")=List of 2
  .. ..$ : chr [1:5] "1" "2" "3" "4" ...
  .. ..$ : chr [1:2] "average_rating" "ratings_count"
 $ totss       : num 1.75e+14
 $ withinss    : num [1:5] 2.83e+12 2.39e+12 2.43e+12 2.75e+12 7.97e+11
 $ tot.withinss: num 1.12e+13
 $ betweenss   : num 1.64e+14
 $ size        : int [1:5] 23 13257 357 75 2
 $ iter        : int 6
 $ ifault      : int 0
 - attr(*, "class")= chr "kmeans"

fviz_cluster(final_kmeans1, data = goodreads_kmeans1)

We can see that the data are clearly classified into 5 different clusters. These clusters can be ordered by ratings count (i.e, the mean ratings count of these clusters can be ordered in a increasing order with clear differences).

As the rating count decreases, the average rating are more widely spread (i.e the average rating rate is higher), which means that the average rating is neither concentrated nor correct.

We also can extract the clusters and add to our initial data for futher analysis at the cluster level:

goodreads_kmeans1_result <- goodreads %>%
                      mutate(Cluster = final_kmeans1$cluster)
goodreads_kmeans1_result

Now, we will try to apply K-means algorithm for all of the numerical variables of this dataset.

goodreads_kmeans2 <- goodreads[,c('average_rating','num_pages','ratings_count','text_reviews_count')]
set.seed(123)
fviz_nbclust(goodreads_kmeans2, kmeans, method = "wss")

fviz_nbclust(goodreads_kmeans2, kmeans, method = "silhouette")

After considering the elbow plot and sihouette plot, we once again come up with the optimal number of clusters of 5. As a result, we will again cluster our dataset into 5 groups.

set.seed(123)
final_kmeans2 <- kmeans(goodreads_kmeans2, 5, nstart = 25)
print(final_kmeans2)

K-means clustering with 5 clusters of sizes 2, 75, 13257, 23, 357

Cluster means:
  average_rating num_pages ratings_count text_reviews_count
1       4.030000  409.0000   4998636.500         82004.5000
2       4.027733  395.6133    667374.147         15690.9467
3       3.927476  340.7500      5645.072           239.9849
4       4.100000  409.2174   1760033.261         32802.3478
5       4.015490  387.9188    191223.644          5718.3950

Clustering vector:
   1    2    3    4    5    6    7    8    9   10   11   12   13   14   15   16   17   18   19   20   21   22   23   24   25   26   27   28   29 
   4    4    1    3    4    3    3    3    3    5    3    3    3    5    3    3    3    3    3    3    3    3    3    3    3    4    3    3    3 
  30   31   32   33   34   35   36   37   38   39   40   41   42   43   44   45   46   47   48   49   50   51   52   53   54   55   56   57   58 
   3    3    3    5    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3 
  59   60   61   62   63   64   65   66   67   68   69   70   71   72   73   74   75   76   77   78   79   80   81   82   83   84   85   86   87 
   3    3    3    3    3    3    5    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3 
  88   89   90   91   92   93   94   95   96   97   98   99  100  101  102  103  104  105  106  107  108  109  110  111  112  113  114  115  116 
   3    3    3    3    3    3    3    3    5    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3 
 117  118  119  120  121  122  123  124  125  126  127  128  129  130  131  132  133  134  135  136  137  138  139  140  141  142  143  144  145 
   3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3 
 146  147  148  149  150  151  152  153  154  155  156  157  158  159  160  161  162  163  164  165  166  167  168  169  170  171  172  173  174 
   5    3    3    3    3    3    3    3    2    3    3    3    3    3    3    3    5    3    3    3    5    3    3    3    3    3    3    3    3 
 175  176  177  178  179  180  181  182  183  184  185  186  187  188  189  190  191  192  193  194  195  196  197  198  199  200  201  202  203 
   3    3    5    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3 
 204  205  206  207  208  209  210  211  212  213  214  215  216  217  218  219  220  221  222  223  224  225  226  227  228  229  230  231  232 
   3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3 
 233  234  235  236  237  238  239  240  241  242  243  244  245  246  247  248  249  250  251  252  253  254  255  256  257  258  259  260  261 
   3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    5    3    3    3    3    3    5    3    3    5    3    3 
 262  263  264  265  266  267  268  269  270  271  272  273  274  275  276  277  278  279  280  281  282  283  284  285  286  287  288  289  290 
   3    5    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3 
 291  292  293  294  295  296  297  298  299  300  301  302  303  304  305  306  307  308  309  310  311  312  313  314  315  316  317  318  319 
   3    3    3    3    3    3    3    3    3    3    3    3    3    3    5    3    3    3    3    3    3    3    3    4    3    3    3    5    3 
 320  321  322  323  324  325  326  327  328  329  330  331  332  333  334  335  336  337  338  339  340  341  342  343  344  345  346  347  348 
   3    3    3    3    3    4    3    3    5    3    5    4    3    3    3    3    3    3    3    3    3    4    3    3    4    3    3    3    2 
 349  350  351  352  353  354  355  356  357  358  359  360  361  362  363  364  365  366  367  368  369  370  371  372  373  374  375  376  377 
   3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    5    3    3    3    5    3    3    3    5    3    5    3    3    3 
 378  379  380  381  382  383  384  385  386  387  388  389  390  391  392  393  394  395  396  397  398  399  400  401  402  403  404  405  406 
   3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    2    3    3    3    3    3    3    3    3    5    3    3    5 
 407  408  409  410  411  412  413  414  415  416  417  418  419  420  421  422  423  424  425  426  427  428  429  430  431  432  433  434  435 
   3    3    3    3    3    3    3    5    3    3    3    3    3    5    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3 
 436  437  438  439  440  441  442  443  444  445  446  447  448  449  450  451  452  453  454  455  456  457  458  459  460  461  462  463  464 
   3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    5    3    3    3    3    3    3 
 465  466  467  468  469  470  471  472  473  474  475  476  477  478  479  480  481  482  483  484  485  486  487  488  489  490  491  492  493 
   2    3    3    3    3    3    3    3    3    3    3    3    3    2    3    3    3    3    3    3    3    3    5    5    3    3    3    3    3 
 494  495  496  497  498  499  500  501  502  503  504  505  506  507  508  509  510  511  512  513  514  515  516  517  518  519  520  521  522 
   3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3 
 523  524  525  526  527  528  529  530  531  532  533  534  535  536  537  538  539  540  541  542  543  544  545  546  547  548  549  550  551 
   3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    5    3    3 
 552  553  554  555  556  557  558  559  560  561  562  563  564  565  566  567  568  569  570  571  572  573  574  575  576  577  578  579  580 
   3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    2    2    3    5    3    5    3    3 
 581  582  583  584  585  586  587  588  589  590  591  592  593  594  595  596  597  598  599  600  601  602  603  604  605  606  607  608  609 
   3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3 
 610  611  612  613  614  615  616  617  618  619  620  621  622  623  624  625  626  627  628  629  630  631  632  633  634  635  636  637  638 
   3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    5    3    2    3    3 
 639  640  641  642  643  644  645  646  647  648  649  650  651  652  653  654  655  656  657  658  659  660  661  662  663  664  665  666  667 
   3    3    5    3    3    5    3    3    3    3    3    3    3    3    3    3    3    3    3    5    3    3    3    3    3    3    4    3    3 
 668  669  670  671  672  673  674  675  676  677  678  679  680  681  682  683  684  685  686  687  688  689  690  691  692  693  694  695  696 
   3    3    3    3    2    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3 
 697  698  699  700  701  702  703  704  705  706  707  708  709  710  711  712  713  714  715  716  717  718  719  720  721  722  723  724  725 
   3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3 
 726  727  728  729  730  731  732  733  734  735  736  737  738  739  740  741  742  743  744  745  746  747  748  749  750  751  752  753  754 
   3    3    3    3    3    3    3    3    3    3    3    3    3    5    3    3    3    3    3    3    3    3    3    3    3    3    3    3    5 
 755  756  757  758  759  760  761  762  763  764  765  766  767  768  769  770  771  772  773  774  775  776  777  778  779  780  781  782  783 
   3    2    3    3    3    3    5    3    3    3    3    3    3    3    2    5    3    5    3    3    3    3    3    3    3    3    3    3    3 
 784  785  786  787  788  789  790  791  792  793  794  795  796  797  798  799  800  801  802  803  804  805  806  807  808  809  810  811  812 
   3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3 
 813  814  815  816  817  818  819  820  821  822  823  824  825  826  827  828  829  830  831  832  833  834  835  836  837  838  839  840  841 
   3    3    3    3    3    3    3    3    3    3    3    5    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3 
 842  843  844  845  846  847  848  849  850  851  852  853  854  855  856  857  858  859  860  861  862  863  864  865  866  867  868  869  870 
   3    3    3    3    3    3    3    5    3    3    3    3    3    3    3    3    3    3    3    5    3    3    3    3    3    3    3    3    3 
 871  872  873  874  875  876  877  878  879  880  881  882  883  884  885  886  887  888  889  890  891  892  893  894  895  896  897  898  899 
   3    3    3    3    5    3    3    3    3    3    3    3    3    3    3    3    3    2    3    3    3    2    3    3    3    3    3    3    3 
 900  901  902  903  904  905  906  907  908  909  910  911  912  913  914  915  916  917  918  919  920  921  922  923  924  925  926  927  928 
   3    3    3    3    3    3    3    3    3    3    3    5    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3 
 929  930  931  932  933  934  935  936  937  938  939  940  941  942  943  944  945  946  947  948  949  950  951  952  953  954  955  956  957 
   3    5    3    3    3    3    3    3    3    5    3    3    3    3    5    3    3    3    3    3    3    3    3    3    3    3    3    3    3 
 958  959  960  961  962  963  964  965  966  967  968  969  970  971  972  973  974  975  976  977  978  979  980  981  982  983  984  985  986 
   3    3    3    3    5    3    3    3    3    3    3    3    3    3    2    3    3    3    3    3    3    3    3    3    5    3    3    3    3 
 987  988  989  990  991  992  993  994  995  996  997  998  999 1000 
   3    3    3    3    3    3    3    3    3    3    3    3    3    3 
 [ reached getOption("max.print") -- omitted 12714 entries ]

Within cluster sum of squares by cluster:
[1] 7.973378e+11 2.754288e+12 2.390668e+12 2.840440e+12 2.436637e+12
 (between_SS / total_SS =  93.6 %)

Available components:

[1] "cluster"      "centers"      "totss"        "withinss"     "tot.withinss" "betweenss"    "size"         "iter"         "ifault"      

fviz_cluster(final_kmeans2, data = goodreads_kmeans2)

Surprising enough, we receive the exactly same result as we do with only two variables, with every observation being in the same group as the first model.

Web scrapping to insert year data for books/series

As we have has ISBNs associated with books, we can retrieve the year when the book was written by web-scrapping from the website as follow.

goodreads_year <- goodreads
goodreads_year$Year <- 0
for (i in c(1:100)) {
  url_i <- paste('https://isbndb.com/book/',goodreads[i,'isbn'],sep = "")
  webpage_i <- read_html(url_i)
  year_data_html <- html_nodes(webpage_i,'td')
  year_data <- html_text(year_data_html)
  goodreads_year[i,'Year'] <- as.numeric(year_data[6])
}
goodreads_year <- goodreads[goodreads$Year >= 1900,]
goodreads_year <- na.omit(goodreads_year)
goodreads_year

However, this code in R may take a little while to code. As a result, we will proceed with a script in Python, takean from the “Goodreads: Analysis and Recommending Books” notebook on Kaggle.

We can easily see some NAs appearing in the column “Year”. This is due to the fact that the website we are scrapping data from lacks the information “year”. In order to further analyze this dataset with the variable “Year”, we omit all of NAs.

However, instead of doing web-scrapping with R, we decided to use a much more effective tool, Python. The Python code for webscrapping can be found within the same file of this notebook. After running the Python script, we save the csv file as goodreads_year.csv, with encoding “UTF-16”. We import the file as follow.

goodreads_year <- read.csv('/Users/ngohoanganh/Desktop/Goodreads Kaggle project/goodreads_year.csv', stringsAsFactor = FALSE, fileEncoding = "UTF-16")
str(goodreads_year)

'data.frame':   13013 obs. of  11 variables:
 $ bookID            : int  1 2 3 4 5 8 9 10 12 13 ...
 $ title             : chr  "Harry Potter and the Half-Blood Prince (Harry Potter  #6)" "Harry Potter and the Order of the Phoenix (Harry Potter  #5)" "Harry Potter and the Sorcerer's Stone (Harry Potter  #1)" "Harry Potter and the Chamber of Secrets (Harry Potter  #2)" ...
 $ authors           : chr  "J.K. Rowling-Mary GrandPré" "J.K. Rowling-Mary GrandPré" "J.K. Rowling-Mary GrandPré" "J.K. Rowling" ...
 $ average_rating    : num  4.56 4.49 4.47 4.41 4.55 4.78 3.69 4.73 4.38 4.38 ...
 $ isbn              : chr  "0439785960" "0439358078" "0439554934" "0439554896" ...
 $ isbn13            : num  9.78e+12 9.78e+12 9.78e+12 9.78e+12 9.78e+12 ...
 $ language_code     : chr  "eng" "eng" "eng" "eng" ...
 $ X..num_pages      : int  652 870 320 352 435 2690 152 3342 815 815 ...
 $ ratings_count     : int  1944099 1996446 5629932 6267 2149872 38872 18 27410 3602 240189 ...
 $ text_reviews_count: int  26249 27613 70390 272 33964 154 1 820 258 3954 ...
 $ Year              : int  2006 2004 1997 2003 2004 2004 2005 2005 2005 2002 ...

print(paste('There are',nrow(goodreads_year),'books whose `Year` attribute can be retrieved, which accounts for',round(nrow(goodreads_year)/nrow(goodreads)*100, digits = 2),'eprcent of the original dataset.'))

[1] "There are 13013 books whose `Year` attribute can be retrieved, which accounts for 94.89 eprcent of the original dataset."

After retrieving the Year variable, we will proceed to analyse the new dataset with the following graphs:

• Average of average_rating for each year
• Total number of books for each year
• Total ratings count and average ratings count for each book of each year
• Total text reviews count and average text reviews count of each book for each year

Average of average_rating of all books for each year

mean_year <- aggregate(average_rating ~ Year, data = goodreads_year, FUN = mean)
mean_year

ggplot(data = mean_year, aes(x = Year, y = average_rating)) +
  geom_line(color = "#FC4E07", size = 1) +
  labs(x = "Year", y = "Average rating", title = "Average rating for each year") + 
  geom_smooth(method = "lm")

goodreads_year[goodreads_year$Year == 1922,]

goodreads_year[goodreads_year$Year == 1931,]

From the graph, we can see two opposite peaks appearing:

• Maximum peak: The average ratings of books written in 1922 reached the peak of 5.0. After re-checking with the data retrieved, there is only one book written in 1922. Moreover, this book received 0 rating and 0 text review.
• Minimum peak: The average ratings of books reached its lowest in 1931 with the average of 2.75, then gradually recover until 1940. As expected, in 1931, there is only one book written, with 4 ratings and 0 text review.

Apart from the peaks mentioned above, there is no other abnormal points within the graph. For the remaining years, the average rating fluctuate around 4.0.

Total books of each year

total_books_year <- goodreads_year %>% count(Year)
names(total_books_year) <- c('Year','number_of_books')
total_books_year

ggplot(data = total_books_year, aes(x = Year, y = number_of_books)) + 
  geom_line(color = "#00AFBB", size = 1) +
  labs(x = "Year", y = "Total number of books", title = "Number of books written for each year") + 
  geom_smooth(method = "lm")

We can easily see that, before 1975, there are not many books written in each year. However, after that timestamp, the books published every year start to increase gradually, and skyrocketted between 1990 and 2006. After reaching its peak, the number of books suddenly dropped significantly, and since 2016, there are less than 10 new books written each year.

Total ratings count and average ratings count for each book of each year

sum_ratings_count_year <- aggregate(ratings_count ~ Year, data = goodreads_year, FUN = sum)
sum_ratings_count_year

ggplot(data = sum_ratings_count_year, aes(x = Year, y = ratings_count)) +
  geom_line(color = "purple", size = 1) +
  labs(x = "Year", y = "Total ratings count", title = "Total ratings count of all books for each year") + 
  geom_smooth(method = "lm")

average_ratings_count_year <- aggregate(ratings_count ~ Year, data = goodreads_year, FUN = mean)
average_ratings_count_year

ggplot(data = average_ratings_count_year, aes(x = Year, y = ratings_count)) +
  geom_line(color = "purple", size = 1) +
  labs(x = "Year", y = "Average ratings count", title = "Average ratings count of each book for each year") + 
  geom_smooth(method = "lm")

The total number of ratings each year seems to follow the pattern of the number of books published each year. However, the average ratings count for each book each year does not. We can see two greatest peaks of around 60,000 ratings in 1952 and 2019, along with various other smaller peaks. The average does not seem to follow any pattern.

Total text reviews count and average text reviews count for each book of each year

sum_text_reviews_count_year <- aggregate(text_reviews_count ~ Year, data = goodreads_year, FUN = sum)
sum_text_reviews_count_year

ggplot(data = sum_text_reviews_count_year, aes(x = Year, y = text_reviews_count)) +
  geom_line(color = "#9ACD32", size = 1) +
  labs(x = "Year", y = "Total ratings count", title = "Total text reviews count of all books for each year") + 
  geom_smooth(method = "lm")

average_text_reviews_count_year <- aggregate(text_reviews_count ~ Year, data = goodreads_year, FUN = mean)
average_text_reviews_count_year

ggplot(data = average_text_reviews_count_year, aes(x = Year, y = text_reviews_count)) +
  geom_line(color = "#9ACD32", size = 1) +
  labs(x = "Year", y = "Average text reviews count", title = "Average text reviews count of each book for each year") +
  geom_smooth(method = "lm")

Once again, the total number of text reviews count each year seems to follow the pattern of the number of books written each year. The average reviews count for each book each year reaches it peak in 2019, also along with some other smaller peaks. This time, we can observe that there is an overall increasing trend with respect to time, which means that as time goes, people are leaving more and more text reviews.